Comments for MEDB 5510, Week 09

Topics to be covered

What you will learn
- Internal validity
- External validity
- Measurement validity
- Five case studies
- Three dichotomies of measurement
- Dichotomy examples
- Split half reliability
- KR-20 and Cronbach’s alpha

Three types of validity

Internal validity
- “The extent to which we can infer that the independent variable caused the dependent variable.”
External validity
- “The extent to which the findings will generalize to other populations, settings, measures, and treatments.”
Measurement validity
- “The quality of accuracy of individual measures or scores. The extent to which a score measures what it was intended to measure.”
- Also measurement reliability

Internal Validity

“The extent to which we can infer that the independent variable caused the dependent variable.”
Three criteria for causality
- IV must precede the outcome variable
- IV must be related to the outcome
- There must be no other variables that could explain why the IV is related to the outcome

Establishing internal validity by the research approach

Hierachy
- Randomized Experiments
- Quasi-Experimental studies
- Comparative
- Associational
- Descriptive

Internal Validity is equivalence and control

Evaluating the internal validity of a study –
- Equivalence of the groups on participant characteristics
- Control of extraneous experiences and environmental variables

Establishing equivalence

Are groups equivalent prior to introduction of IV?
- Assured without further work in randomized studies
- Empirical comparisons for non-randomized studies
- Can you use matching or statistical adjustments?

Establishing control

Extraneous and environmental variables
- Not of direct interest
- Influence the outcome
- Imbalanced
Example: contamination
Is one group affected more than the other?
- Less of an issue for laboratory studies

Other threats to internal validity

Regression to the mean
Dropouts/attrition
Bias in assignment
Carryover effects
Changes in environment
Instrument or observer inconsistency
Patient expectations
Observer bias

Break #1

What you have learned
- Internal validity
What’s coming next
- External validity

External Validity

Population external validity
- Representative sample
- Few or no restrictions
- Few or no dropouts
Ecological external validity
- Naturalness
  - Setting
  - Procedures

Distinction between internal and external validity

Sampling process influences external validity
- Random samples versus non-random samples
Treatment allocation influences internal validity
- Randomized design versus non-randomized design

Trade-offs between internal and external validity

High degree of control
- Avoid issues of equivalence
- Unnatural setting
Low degree of control
- More chances of contamination
- Closer to how medicine is practiced.

Break #2

What you have learned
- External validity
What’s coming next
- Measurement validity

Measurement quotes (1 of 2)

“The government is extremely fond of amassing great quantities of statistics. These are raised to the Nth degree, the cube roots are extracted, and the results are arranged into elaborate and impressive displays. What must be kept ever in mind, however, is that in every case, the figures are first put down by a village watchman, and he puts down anything he damn well pleases.”
- Sir Josiah Stamp, as quoted on Quotetab.

Measurement quotes (2 of 2)

“only scientists are arrogant enough to think that they always observe with rigorous and objective scrutiny”
- Stephen Jay Gould, The Mismeasure of Man, page 36.

Measurements that warrant closer scrutiny

Patient reported outcomes
- Participant report
Researcher evaluations
- Only when concerned about subjectivity
Psychological constructs
Composite scores

For better or for worse, researchers tend to focus greater attention on certain types of measurements. There’s no hard and fast rule here, but issues of measurement quality tend to appear most often in certain areas.

Don’t think that if your measure is not on this list that it doesn’t deserve careful scrutiny. There’s really no consensus in the research community on what measurements require this extra level of attention.

I think it is a bit unfair, but there is a lot of distrust of patient reported outcomes among researchers. Why not believe what the patient says about himself or herself? Part of it might be that a patient’s answers could potentially be influenced by their mood.

There is also a belief that patient reported outcome measures vary too much from one individual to another. Some people are stoic to a fault and others will complain endlessly at the drop of a hat.

It is worth noting that these factors also influence researcher observation, but researchers don’t like it when you point this out to them. It’s mostly a good thing that researchers require a high level of scrutiny of patient reported outcomes, but perhaps other measures deserve just as high a level of scrutiny.

There is a fair amount of scrutiny of researcher evaluations when these evaluations are perceived as having a high level of subjectivity. Now, our perceptions as to what is subjective are also subjective, so you need to be careful.

Psychological constructs are tools used to measure aspects of human behavior, such as intelligence, self-esteem, stress, and extraversion. In spite of recent advances in brain imaging, you cannot, for the most part, peek inside someone’s mind and understand how they think.

Finally, many measures in clinical research are composites of one or more items. These individual items are scored and added up to get a total. If the individual items are chosen well, this can be a very effective approach, but you need to be careful.

Measurement Reliability

Synoynms: consistency, precision, stability
No measurement is perfectly reliable
Dependent on the population
Look for prior efforts in reliability

When you measure something, you want that measurement to be consistent, precise, and stable. You don’t want something that changes as the phases of the moon change. You don’t want a measurement that changes depending on who the attending physician is. You don’t want a measurement that changes depending on any environmental factors that are extraneous to what you are measuring.

If your measure is not stable, then you have difficulty in assessing whether a change in that measurement is due to your intervention or due to the phases of the moon.

One thing you need to keep in mind is that the reliability coefficient is dependent on the population it is based on. Your book doesn’t mention this, but it is important. Change the population and you change the reliability coefficient. Something with a great reliability coefficient in a population of college students might be terrible in a population with limited literacy skills, for example.

Since no measurement is ever conducted without some Reliability is usually established when a measure is developed. When you go about using a measure, look at what’s already been published. Make sure it used in a context similar to yours. It’s a whole lot easier to find a measurement that is already proven to be reliable than to develop your own measure and then establish its reliability.

Measurement Validity, 1

Reliability by itself is not enough.
- Consistent measures of the “wrong thing” is bad
Examples of the wrong thing
- Measuring anxiety instead of stress
- Measuring transient changes in a patient’s mood rather than chronic depression

Measurement Validity, 2

Validity
- “Degree to which a measure … measures that which it was intended to measure”
Reliability is a pre-requisite for validity
Validity is a journey and not a destination

Break #3

What you have learned
- Measurement validity
What’s coming next
- Five case studies

Case study #1 - Neighborhood Environment Survey

The image on the left is a picture of the Boynton neighborhood in Detroit, Michigan. It was downloaded from

https://en.wikipedia.org/wiki/File:Boynton_neighborhood_Detroit_(Annabelle_Street).jpg

and is published under a public domain license.

The image on the right is a screenshot from the publication

Crum RM, Lillie-Blanton M, Anthony JC. Neighborhood environment and opportunity to use cocaine and other drugs in late childhood and early adolescence. Drug and Alcohol Dependence 43 (1996) 155-161.

It is copyrighted and made available here through the “Fair Use” provisions of copyright law. Your use of this image may or may not fall under the Fair Use provisions.

This is the Neighborhood Environment Survey.

I realize this image might be difficult to read on your computer. I’m going to magnify this in a bit, but notice that this is a series of 18 true/false questions.

Case study #1 - Neighborhod Environment Survey

Case study #2 - Pain scale

Case study #3 - Apgar score

Case study #4 - Boston Bowel Prep Score

The image on the left is a colonoscope. It was downloeaded from

https://commons.wikimedia.org/wiki/File:PENTAX_Colonoscope001.jpg

and is published under an open source license.

On the right is the first page of a publication available through Pubmed Central. It is also available under an open source license.

This an except from an article establishing the reliability and validity of the Boston Bowel Prep score. When you get a colonoscopy done, you’re supposed fast for a full day and also drink a foul tasting preparation. Almost a gallon of this stuff. When you’re done, the next day, your bowel is cleaned out enough that the colonoscopist can go hunting for polyps in your colon.

An objective measure of how well you did with your fasting and with your drink preparation is the Boston Bowel Prep Score. This is an excerpt from the research article that established reliability and validity of this measure.

Lai EJ, Calderwood AH, Doros G, Fix OK, Jacobson BC. The Boston bowel preparation scale: a valid and reliable instrument for colonoscopy-oriented research. Gastrointest Endosc. 2009 Mar;69(3 Pt 2):620-5. doi: 10.1016/j.gie.2008.05.057. Epub 2009 Jan 10. PubMed PMID: 19136102; PubMed Central PMCID: PMC2763922.

Case study #4 - Boston Bowel Prep Score

Excerpt from Lai et al article

Case study #5 - Disgust Scale Revised

Case study #5 - Disgust scale revised

Break #4

What you have learned
- Five case studies
What’s coming next
- Three dichotomies of measurement

First dichotomy: Who is the measurer?

Self reported outcomes
- Also know as patient reported outcomes
Researcher evaluations
- Only when concerned about subjectivity

So far, you’ve seen five case studies where you may want to assess reliability and validity. These are very different types of measurements and you can classify them into three important dichotomies. These three dichotomies will help you assess the best approach to take for assessing reliability and validity.

I think it is a bit unfair, but there is a lot of distrust of self reported outcomes among researchers. Why not believe what the patient says about himself or herself? Part of it might be that a patient’s answers could potentially be influenced by their mood. They might also be influenced by a desire to look good. They may want to give an answer that they think the interviewer wants to hear.

There is also a belief that reported outcome measures vary too much from one individual to another. Some people are stoic to a fault and others will complain endlessly at the drop of a hat.

It is worth noting that these factors also influence researcher observation, but researchers don’t like it when you point this out to them. It’s mostly a good thing that researchers require a high level of scrutiny of reported outcomes, but perhaps other measures deserve just as high a level of scrutiny.

Second dichotomy: How many pieces?

Composite scores
- Sum or average
Single measurement

Many measures in research are composites of one or more items. These individual items are scored and added up to get a total. Sometimes you average them instead of summing them. That’s a minor distinction. In general, a composite measure involves evaluating something from many different angles. If you do this well,, this can be a very effective approach. But you do need to be careful.

In contrast, some measures are just a single number and there is no summing or averaging involved. There is just a single question or a single evaluation. It is a difficult decision to choose whether to develop a measurement using a composite scale versus a single measurement, and it depends a lot on what you are measuring.

The general consensus among researchers is that you get a better picture of something with a composite scale, but some measures do not warrant this extra effort.

Third dichotomy: is the measure soft or hard? 1

Psychological or social constructs
- Created and accepted by you and me
- Impossible to observe directly
- Examples: stress, anxiety

Third dichotomy: is the measure hard or soft? 2

Biological or physical measure
- Has an objective reality
- Potential for direct observation
- Example: obesity, dementia

Break #5

What you have learned
- Three dichotomies of measurement
What’s coming next
- Dichotomy examples

Break #5

What have you learned
- Dichotomies of measurement
What’s coming next
- Dichotomy examples

Examples of self reported outcomes

Examples of researcher evaluation

Examples of composite scores

Examples of single measurements

Examples of constructs

Examples of biological or physical measurements

Pop quiz

Is the disgust score
- a self report or a researcher evaluation?
- a composite measure or a single measurement?
- a psychological or social construct or a biological or physical measurement?

Break #6

What you have learned
- Dichotomy examples
What’s coming next
- Split half reliability

Measurement Reliability

Synoynms: consistency, precision, stability
Classical test theory
- Observed value = True value + Measurement error
- This is a purely hypothetical model
Reliability coefficient
- Variance of true values / Variance of measured values
Depends on your population

When you measure something, you want that measurement to be consistent, precise, and stable. You don’t want something that changes as the phases of the moon change. You don’t want a measurement that changes depending on who the attending physician is. You don’t want a measurement that changes depending on any environmental factors that are extraneous to what you are measuring.

If your measure is not stable, then you have difficulty in assessing whether a change in that measurement is due to your intervention or due to the phases of the moon.

Most measures of reliability rely on the true value model. This model says that the observed value of a measurement is equal to the true value plus measurement error. A measurement is reliable if the measurement error is small. Since the true value is almost always unknown, it is only a hypothetical model.

Your book talks about a reliability coefficient which is the variance of the true scores in a population divided by the variance of the observed scores in a population. Measurement error guarantees that the numerator is always less than or equal to the denominator. The reliability coefficient is equal to one only if there is no measurement error.

You should not be too surprised to find out that the reliability coefficient is a hypothetical value and can never be measured directly. But there are several indirect approaches.

One thing you need to keep in mind is that the reliability coefficient is dependent on the population it is based on. This is very important. Change the population and you change the reliability coefficient. Something with a great reliability coefficient in a population of college students might be terrible in a population with limited literacy skills, for example.

Measurement Reliability

No measurement is perfectly reliable
- Strive for 0.7 or higher in research
- 0.6 is “borderline”.
- Might require 0.9 or higher for individual decisions

Since no measurement is ever conducted without some measurement error, no measurement has perfect reliability. You need to make a value judgement about whether the deviation from the truth is small enough that you can safely ignore it.

There are some informal standards for reliability. These choices can seem a bit arbitrary, but they are fairly well accepted in the research community.

In order for a measurement to be reliable enough to use in a research setting, where you are trying to characterize how a group of people are affected by an intervention, you would like a reliability coefficient of 0.7 or higher. It’s not perfect, but the individual measurement errors would be averaged out when you compute group means.

But if you are making decisions that might affect an individual, then you’d want a much higher level of reliability. Individual decisions might involve acceptance into a training program, for example. You would hate to see a large measurement error dominate the decision about an individual. In these settings, a reliability coefficient of 0.9 or higher might be asked for.

For the record, some sources say that your reliability could go as low as 0.6 and still be okay. Other sources disagree. If you have such a value, go ahead and report it using a term like “borderline” or “marginal” and hope that your peer-reviewer isn’t a stickler for this sort of thing.

Reliability is usually established when a measure is developed. When you go about using a measure, look at what’s already been published. Make sure it used in a context similar to yours. It’s a whole lot easier to find a measurement that is already proven to be reliable than to develop your own measure and then establish its reliability.

Indirect measures of the reliability coefficient

Test-retest
Interrater
Internal consistency

Parallel forms

“No man ever steps in the same river twice, for it’s not the same river and he’s not the same man.”
- Heraclitus
Used when you can’t run the same measurement twice.
How to develop parallel forms
- Change the question order
- Minor changes to the wording
Difficult to develop two parallel forms of the same measurement.

Sometimes the very act of measuring someone changes that person. I do this all the time. I put a quiz up each week, not to test you so much as to reinforce some of the key messages in my videos. The questions are not intended to challenge you and assess how much you’ve learned. Having come up with an answer, that helps you remember the key concepts better.

The opposite tendency can occur as well. The novelty of answering questions wears off over time and people may grow tired or bored and not answer the exact same questions a second time.

How likely is this to happen? It depends a lot on what is being measured. Measures of knowledge and understanding are more likely to have carry over effects.

In some settings, you can create a second version of your measurement by making minor changes. This could be in the wording or the ordering of the questions.

How much of a change do you want? Too little and you still have problems with carry over. Too much and you are no longer measuring the same thing.

The parallel forms measure of reliability is not used that frequently, because it just about kills you to get one version of a measurement up and running. Who wants to develop two parallel forms. It’s worth introducing here, though, because it helps you understand the next three forms of reliability.

Split half reliability

Only used for composite measurements
Split into halves, correlated
- Odd-even split
- Random split
Brown-Spearman adjustment

If your measurement is a composite measure, then you can look at the correlation of the individual components to assess reliability.

You could split the measure in half, calling the even numbered items the first form and the odd numbered items the second form. The correlation between the odds and the evens is a measure of reliability.

It doesn’t have to be evens versus odds. You might want to assign items randomly to the first half versus the second half.

You do need to be careful, though. The reliability of a composite measurement is frequently thought to be related to the number of items in the composite. The greater the number of items, the greater the reliability. So if you artificially shorten the measurement, you are underestimating reliability. There is a simple adjustment, called the Spearman-Brown formula that most researchers use when looking at split half correlations.

Break #7

What you have learned
- Split half reliability
What’s coming next
- KR-20 and Cronbach’s alpha

Kuder-Richardson 20 formula

Only for composite measures with binary items
Book’s formula is confusing
- \(S^2\) and \(\sigma^2\) used interchangably
- Subscripts are missing
Correct formula
- \(X_i=\Sigma_i B_{ij}\)
- Where \(B_{ij}\) is binary (0 or 1)
- \(KR-20=\frac{1}{I-1}\big(1-\frac{\Sigma_i \hat{p}_i\hat{q}_i}{S^2}\big)\)
- where \(S^2=\Sigma_i(X_i-\bar{X})^2\),
- \(\hat{p}_j=\Sigma_i B_{ij}\),
- \(\hat{q}_j=\Sigma_i (1-B_{ij})\)

Kuder-Richardson 20 interpretation

KR-20
- \(\frac{1}{I-1}\big(1-\frac{\Sigma_i \hat{p}_i\hat{q}_i}{S^2}\big)\)
- \(\Sigma_i \hat{p}_i\hat{q}_i\) is a theoretical minimum variation
- \(S^2\) is observed variation
- \(S^2 = \Sigma_i \hat{p}_i\hat{q}_i\) implies randomness
- \(S^2 > \Sigma_i \hat{p}_i\hat{q}_i\) implies internal consistency

If you are curious, the formula is comparing a theoretical minimum variation, a variation computed using independent Bernoulli random variables, but with different p’s and q’s for each item. Strictly speaking, this is not accurate, but a smaller value than the sum of the pq’s could only occur if there is negative correlation among the individual items, and this implies almost a conspiracy among the individual items to make things as bad as possible for you.

You compare this to the variation observed among the total scores in the sample. If the observed variation is equal to the theoretical minimum, the individual items are behaving randomly, with no internal consistency. This means that any split halves that you could compute would have next to no correlation.

If there is much more observed variation, that means that people show positive correlations. Low on one item means low on most of the other items and high on one item means high on the other items. This positive correlation is a measure of internal consistency.

Keep in mind that if you have a different population, the minimum variation would stay the same, but the observed variation might change. So a measure that is reliable in one population might prove to be not reliable in a different population.

Cronbach’s alpha, formula

Used for composite measurements with continuous items
Book’s formula is confusing
- \(\Sigma S^2\) should be \(\Sigma S_i^2\)
Correct formula
- \(X_i=\Sigma_j X_{ij}\)
- \(\alpha=\frac{1}{I-1}\Big(1-\frac{\Sigma_i S_i^2}{S^2}\Big)\)
- where \(S_i^2=\Sigma_j(X_{ij}-\bar{X}_j)^2\), and
- \(S^2=\Sigma_i(X_i-\bar{X})^2\), and

Cronbach’s alpha, interpretation

Cronbach’s \(\alpha\)
- \(\frac{1}{I-1}\Big(1-\frac{\Sigma_i S_i^2}{S^2}\Big)\)
- \(\Sigma_i S_i^2\) is a theoretical minimum variation
- \(S^2\) is observed variation
- \(S^2 = \Sigma_i S_i^2\) implies randomness
- \(S^2 > \Sigma_i S_i^2\) implies internal consistency
Cronbach’s alpha is NOT a measure of unidimensionality

Just like Kuder-Richardson 20, Cronbach’s alpha computes a a theoretical minimum variation. This time it is a sum of the variances for the individual items. Strictly speaking, this is not accurate, but a smaller value than the sum of the variances implies a deep and dark conspiracy against you by the individual items in your composite.

You compare this to the variation observed in the total score. If the two values are close, that tells you that the individual items are more or less independent of each other and that any split halves that you might compute would have little or no correlation.

If there is much more observed variation, that means that people show positive correlations. Low on one item means low on most of the other items and high on one item means high on the other items. This positive correlation is a measure of internal consistency.

Some people confuse the concept of internal consistency with uni-dimensionality. Uni-dimensionality means that all of the items are measuring the same construct. If they are, then Cronbach’s alpha will be large. But you can also get a large value for Cronbach’s alpha, even when the items are measuring multiple constructs, especially when you have a large number of items. Dimensionality can only be measured using some form of factor analysis.

Break #8

What you have learned
- KR-20 and Cronbach’s alpha
What’s coming next
- Test-retest and interrater reliability

Test-retest reliability

Test-retest reliability (also called repeatability)

Correlation of two measurements separated by time
Length of time interval is critical
- No carry-over
- No changes in the true score
Useful for composite scores and single values
Useful for self-report and researcher evaluation
Not possible for some measures

Test-retest reliability is the correlation coefficient of two measurements taken at different times. This is also known as repeatability.

The correlation coefficient between the two measurments is an estimate of the reliability coefficient.

The time interval is critical here. You don’t want two measurements that are so close together that the measurement error for the first measurement is correlated with the measurement error of the second measurement.

Suppose you are measuring your patient’s knowledge about a disease. If you give the same test only a few minutes apart, your patient will remember his/her answers to the first test when answering the second test.

So you want a long enough time interval that there is no carry over effect.

But too large an interval is also problematic. You want to make sure that the true score is the same (or very close)

Over what size interval would you expect the measure to be stable? It depends on what you are measuring. Intelligence is likely to be stable along long time frames but mood changes rapidly.

Test-retest reliability works well for any type of measure. But it can’t always be applied. The Apgar score has to be measured at the time of birth. If the baby is blue, limp, and unresponsive after three hours, that’s not quite the same as the same occuring right after birth.

Inter-rater reliability

Used for researcher evaluations only
Simplest case
- Two independent raters
- Ratings for every patient
Analysis
- Intraclass correlation or Cohen’s Kappa
Extensions

When the researcher does the evaluation and there is concern that a subjective element may creep in and cause measurement error. Your observed score might be higher or lower depending only on who is rating you.

Reliability is pretty simple to measure in this setting, if you have the resources. Just get two raters and have them both compute the measurement. If the correlation between the two raters is high, you have good reliability.

Rather than computing a direct correlation, inter-rater reliability is usually computed as an intra-class correlation. The intraclass correlation generalizes naturally to more complex settings.

If your measurement is binary (note the entire measurement is binary, which is not the same as saying that the individual components of a composite score are binary), then a different statistic, Cohen’s Kappa is used. Like the intraclass correlation, there are extensions of Kappa to multiple raters.

You can’t always have all the raters rate all the patients, especially if you have more than two raters. There are extensions to cases where you have random assignments of patients to different raters, but the formulas are tricky.

Practical guidance on reliability

Is there previous literature?
- Report their reliability coefficients
Is your setting similar?
- Different demographics?
- Different cultural norms?
- Different literacy?
- Different language?
Compare to reliability in your sample
- Test-retest and inter-rater reliability preferred.
- 0.7 or higher

You should include a discussion of reliability in your literature review. Cite the reliability coefficients in previous work, as it adds to the credibility of your proposed research.

But take a step back and ask if you can extrapolate safely to the research setting that you propose. Recall the hypothetical reliability coefficient. It compared the variation of the true score to the variation of the observed score across patients in the population you are studying. If your population is quite different than the populations in your literature review, you have no guarantee that a measurement proven to be reliable in previous work will continue to stay reliable in your setting.

Some differences to look for are differences in the demographics of your population, differences in cultural norms and expectations, differences in literacy levels (especially for measurements that require your patients to read and respond to a questionnaire).

If you are measuring something that requires translation to a different language, keep in mind that not all concepts translate well from one language to another. Sometimes it helps to pay for a second and independent translation back to the original language. If there are discrepancies, then maybe it was in the back-translation, but more likely, you are asking for a different type of information in your new language without realizing it.

If you can, incorporate a measure of reliability into your study. There are two reasons for this. First, your setting may be different enough to raise concerns. Getting a current measure of reliability helps to allay those concerns. Second, reliability is never quite perfect, because all of the measures of reliability are indirect measures. Your effort to assess reliability will supplement the previous work on reliability and make things a bit easier for future researchers.

I have a strong preference for test-restest reliability or inter-rater reliability, if you can get it. The other measures of reliability, parallel forms, the split half correlation, Kuder-Richardson 20, and Cronbach’s alpha are measures of the internal homogeneity of your composite measure. In my mind they are a poor substitute for test-retest reliability or inter-rater reliability.

I do not like these measures. Let me restate that. I despise these measures. They are simplistic and fail to measure what I think are the important features of reliability (stability over time and consistency between raters). I think people use them mindlessly and fail to recognize that they are measuring something very limited.

If you can’t measure reliability using a test-retest approach or using inter-rater reliability, then go ahead and use these other approaches. But they are a pale substitute in my opinion.

The general target value for a reliability coefficient is 0.7 or higher. You might get by with a reliability coefficient of 0.6, but don’t count on it.

Break #9

What you have learned
- Test-retest and interrater reliability
What’s coming next
- Face and content validity

Types of measurement validity

Face validity
Content validity
Response process evidence
Criterion validity
Construct validity

Face validity and content validity

Face validity
- Opinions from your patients
- Subjective and unquantifiable
Only used for composite measures

Content validity
- Opinions from outside experts
- Subjective and unquantifiable
Only used for composite measures

There are varying definitions of face validity and content validity. Let me share the defintions that I like. This is my class and I get to dictate the rules. But I’ll let you know what others define these two terms as.

Face validity is information from your patients, typically for a composite measurement. They look at the individual items in your composite measurement and tell you the ones that don’t really belong. They should also tell you about items that are missing in your composite measurement that you should include. Face validity is a totally subjective approach and to some people it seems like letting the inmates run the asylum.

To be fair, face validity is an important step in establishing validity, but it should probably not be the only step.

Content validity is information from content experts rather than your patients. But otherwise, it is the exact same thing. The experts look at your composite measurement and tell you that certain things need to go and other things need to be added.

Now, who is an expert? It can be anyone, really. Normally, you would use credentials like a degree and a publication record in the area to establish that someone is qualified to tinker with your measurement.

Both face validity and content validity are purely qualitative. There is no numeric measure or score that you get from these types of validity. You do have to establish consensus, if you seek face validity or content validity from more than one source, but this is usually established qualitatively.

There are structured ways to get information about face and content validity from your patients and from an expert panel, such as the Delphi method. You can use these methods, or you could just use a structured interview.

Even though these approaches are soft, they are well worth the effort.

Now some people use the terms face validity and content validity interchangably. Your book says that face validity is just looking at the measure and giving a general impression while content validity requires delving into the individual items of a composite measure.

I won’t say that your book is wrong, but your book is wrong. Actually, I’m probably wrong, but I’m your teacher and you’re stuck with my opinion, at least until the semester ends.

Seriously, if there is a disagreement in the research community about how to establish validity, what you do is you do it your way, but with the expectation that when you submit your paper to peer review, plan for the possibility that the peer reviewer will ask you to define things their way. It’s normally not a good idea to fight a peer-reviewer, especially when there is no consensus in the research community, unless they are asking for something that is seriously wrong and misleading.

Now your teacher, on the other hand, you can argue with him until the cows come home. He actually will enjoy the argument and you won’t get him to shut up about the varying research perspectives.

Question.

Should Statisticians work on problems that are subjective and unquantifiable?

Answer.

Yer darn tootin!

Response process evidence

Observe the process
- Watch as patients fill out the form
- Ask questions along the way
- Monitor response times
- Encourage them to think aloud
Supplement with interview
Goal is to identify problematic elements
- Confusion, misunderstandings, language issues

Response process validity is the direct observation of patients as they fill out the survey that you are using for measurement. You can think of it as part of the face validation, or you can call it an additional type of validation. I like the latter because it sounds more impressive.

There’s nothing too difficult about this. As you observe the process, ask questions, see if there are any items that seem to take too long to answer. Encourage your patients to talk aloud as they are working. If you want to get really fancy, you can use eye tracking to see if someone is losing focus or getting distracted.

You can supplement this process with an interview afterwards. Your goal in this exercise is to identify items that are confusing or ambiguous, or which seem to draw the wrong type of response. Look especially for issues which may come from the use of excessively technical language.

You can do this sort of exercise with experts as well as patients. Ask your experts to pretend that they are patients and get them to fill things out, talk aloud, and ask them questions along the way.

Break #10

What you have learned
- Face and content validity
What’s coming next
- Criterion and construct validity

Criterion validity

Comparison to external criterion
- Represents “truth”
- Not always available
Predictive evidence
- Measurement in the future
- Be careful about dropouts
Concurrent evidence
- Measured at the same time

Criterion validity is the most straightforward approach to establishing validity. You want to see how well your measurement corresponds with what it’s supposed to measure. So include what your supposed to measure and see how strongly it correlates with what you want to measure.

This isn’t always possible, of course, but if you can measure truth then go for it!

Now, why, might you ask yourself, would you use a measurement that correlates well with truth when you can measure truth directly? It probably has something to do with time or money. You can measure the truth but it costs too much to do it in a big study. Or it takes way too long. So you run a smaller study where you measure truth, show that your cheaper and faster measurement correlates well with the truth, and then you can save a whole bunch of time and money in the big study.

Your evidence for validity is predictive evidence if the truth represents something that occurs in the future, meaning after your measurement is taken. In the big study, you can’t wait around and wait for the truth to reveal itself. But in a smaller study, you might have that luxury.

It’s important in using predictive evidence that you don’t have dropouts, especially if those dropouts tend to differ from those who do provide you with data.

Your book offers an interesting example of this with standardized testing for college admission. A school might want to correlate an SAT score, for example, with the grades that a student gets after one year of college. Easy to do, but think about the dropouts. A college, for the most part, is going to admit only those people who score above a cutoff for the SAT. You lose information about those who scored low on the SAT and are left only with those students in a narrow range of SAT scores. It’s even worse if the students who score super high on the SAT decide to attend a more prestigious university than your little podunk college.

Another example is using criterion validity for a test intended to diagnose disease. Suppose you have a test that can predict appendicitis. Patients who score high on the measurement, you send them straight to the OR, so you can cut out the appendix before it ruptures. But what about the patients who score low. They probably don’t have appendicitis, but you don’t know. They won’t volunteer to get cut open in the name of science.

Predictive evidence can sometimes take too long, so you may want to use concurrent evidence, evidence that you can collect at the same time as your measurement. Your book suggests that you ask colllege students at the end of their first year to re-take the SAT and see how that re-take correlates with the grades they are just receiving. It’s not perfect, but it certainly takes less time.

The other application of concurrent evidence is when you don’t have a direct measure of truth, but you have an already validated measure of truth that you can collect concurrently with your new measure. The assumption here, as earlier is that your new measure is cheaper or faster than the currently used and validated measure. If you correlate well with an already validated measure, and that validated measure has already been shown to correlate well with the truth, then you have indirectly established criterion validity.

Now this approach has limits. You can never get quite as much evidence of validity as the already validated measurement has.

Construct validity

Used for a psychological construct
No direct measure of the truth exists
Define associations consistent with your constuct
- Does your measurement show the expected association?
- Known as convergent evidence
Define non-associations with your construct
- Does your measurement also show non-association?
- Known as discriminant or divergent evidence

Alternative framework for validity

Content
Response processes
Internal structure
Relations to other variables
Consequences

Validity of diagnostic tests

Sensitivity
- A test’s ability to obtain a positive result when the target condition is really present
Specificity
- A test’s ability to obtain a negative result when the target condition is really absent

Break #11

What you have learned
- Criterion and construct validity
What’s coming next
- Case studies revisited

Case study #1

Case study #1 - Neighborhood environment survey

Reliability - What you can’t do
- Inter-rater reliability
Reliability - What you can do
- Test-retest reliability
- Cronbach’s alpha

Case study #1 - Neighborhood environment survey

Validity - What you can’t do
- Criterion validity
Validity - What you can do
- Face/content validity
- Response process validity
- Factor analysis
- Construct validity

Case study #2

Case study #2 - Pain scale

Reliability - What you can’t do
- Inter-rater reliability
- Cronbach’s alpha
Reliability - What you can do
- Test-retest reliability

Case study #2 - Pain scale

Validity - What you can’t do
- Criterion validity
- Face/content validity
- Response process validity
- Factor analysis
Validity - What you can do
- Construct validity

Case study #3

Case study #3 - Apgar score

Reliability - What you can’t do
- Test-retest reliability
Reliability - What you can do
- Inter-rater reliability
- Cronbach’s alpha

Case study #3 - Apgar score

Validity - What you can’t do
- Criterion validity
Validity - What you can do
- Face/content validity
- Response process validity
- Construct validity

Case study #4

Case study #4 - Boston Bowel Prep Score

Reliability - What you can’t do
- Test-retest reliability
- Cronbach’s alpha
Reliability - What you can do
- Inter-rater reliability

Case study #4 - Boston Bowel Prep Score

Validity - What you can’t do
- Face/content validity
- Response process validity
- Construct validity
Validity - What you can do
- Criterion validity

Case study #5 - Disgust Scale Revised

Case study #5

What do you think?

What measures of reliability?
- Test-retest reliability
- Inter-rater reliability
- Measures of internal consistency

What measures of validity?
- Face/content validity
- Response process validity
- Criterion validity
- Construct validity

Conclusion

What you’ve seen today
- Internal validity
- External validity
- Measurement reliability
- Measurement validity
- Three dichotomies of measurement
- Five case studies